System and methods for improved aerial mapping with aerial vehicles

ABSTRACT

A method for image generation, preferably including: generating a set of mission parameters for a UAV mission of the UAV associated with aerial scanning of a region of interest; controlling the UAV to perform the mission; generating an image subassembly corresponding to the mission; and/or rendering the image subassembly at a display.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S application Ser. No.16/805,415, filed 28 Feb. 2020, which is a continuation of U.S.application Ser. No. 15/887,832, filed 2 Feb. 2018, which claims thebenefit of U.S. Provisional Application Ser. No. 62/453,926, filed on 2Feb. 2017, each of which is incorporated in its entirety by thisreference.

TECHNICAL FIELD

This invention relates generally to the field of unmanned aerialvehicles and more specifically to a new and useful system and methodsfor improved aerial mapping with aerial vehicles.

BRIEF DESCRIPTION OF THE FIGURES

FIG. 1 is a flowchart representation of an embodiment of a method forimproved aerial mapping.

FIG. 2 depicts a variation of a portion of a method for improved aerialmapping.

FIG. 3 is a flowchart representation of a variation of an embodiment ofa method for improved aerial mapping.

FIG. 4 is a schematic of an example of a portion of a method forimproved aerial mapping.

FIG. 5 depicts an example application of a portion of a method forimproved aerial mapping.

FIG. 6 depicts a schematic representation of a feature tracking process.

FIG. 7 is a flowchart representation of a variation of an embodiment ofgenerating the image subassembly.

DESCRIPTION OF THE EMBODIMENTS

The following description of the embodiments of the invention is notintended to limit the invention to these embodiments, but rather toenable any person skilled in the art to make and use this invention.

1. Method

As shown in FIG. 1, a method 100 for improved aerial mapping with anunmanned aerial vehicle (UAV) preferably includes: generating a set ofmission parameters for a mission of the UAV, the mission of the UAVassociated with aerial scanning of a region of interest S110; for eachof a set of one or more subregions of the region of interest, executinga corresponding subportion of the mission with transmission of the UAValong the flight path S120 (e.g., wherein the subportion includes astarting phase, an intermediate phase, and a terminating phase); andduring each subportion of the mission, generating an image subassemblycorresponding to the subportion of the mission S130, combining the imagesubassembly with a running aggregation of images S140 (e.g., anaggregation of image subassemblies associated with other subportions ofthe mission), and rendering the image subassembly and/or runningaggregation of image subassemblies at a display of an electronic deviceof a user S150. In some variations, the method 100 can additionally oralternatively include generating an orthorectified representation of theregion of interest S160; generating a projected map of the region ofinterest S170; and/or generating an analysis from at least one of theorthorectified representation and the projected map of the region ofinterest S180. However, the method 100 can additionally or alternativelyinclude any other suitable elements.

The method 100 can confer several benefits. First, the method can enableorthomosaic and/or point cloud generation using image sets with lessframe overlap (e.g., especially side overlap, but additionally oralternatively front overlap) than needed in traditional methods. Forexample, changes in camera pose between adjacent legs of aboustrophedonic flight path can be robustly determined based on featurematching of temporally-close (e.g., consecutive) frames (and/or onassociated telemetry data) and on point cloud generation performed basedon such feature matching, rather than relying on feature matchingbetween overlapping regions (e.g., overlapped side regions) of imagescaptured during the adjacent legs. Second, the method can reducecomputational processing demands as compared with traditional methodsfor point cloud generation (e.g., via optimizations such as restrictedbundle adjustments and/or compact data representations), therebyenabling rapid (e.g., near real-time) and/or local (e.g., on userdevices, such as tablets and/or phones) processing and informationoutput. Such rapid and/or local processing and output can enable rapiduser feedback, such as enabling modification of the mission (e.g., basedon user requests) while the UAV is still in flight. However, the methodcan additionally or alternatively provide any other suitable benefits.

The method 100 functions to efficiently generate aerial maps of regionsof interest using a single UAV or a fleet of UAVs, and in a manner thataccounts for UAV state, environmental aspects, terrain aspects (e.g.,homogenous terrain, non-homogenous terrain), and/or any other suitableaspects. The method 100 can additionally or alternatively function tofacilitate generation of analyses based upon the aerial maps producedaccording to the method, wherein, in specific examples, the analyses canprovide one or more of: photogrammetric measurements, manipulatableorthomosaic maps, manipulatable models (e.g., 3D models, meshes,point-clouds, etc.), maps or representations annotated/layered withvisualizations of custom or known indices (e.g., according to normalizeddifference Indices, according to enhanced indices, etc.), and any othersuitable output.

The method 100 is preferably implemented using a platform interfacedwith one or more unmanned aerial vehicles (UAVs) and operable to provideservices in relation to one or more of: UAV mission preparation, UAVmission execution, UAV mission modification, aerial image data analysis,aerial video data analysis, analysis of sensor data from aerial systems,aerial telemetry analysis, and/or any other suitable type of UAV-relatedservice. The method(s) 100 can thus be implemented using one or moresystem elements described in U.S. application Ser. No. 14/717,955 filedon 20 May 2015 and titled “Method for Adaptive Mission Execution on anUnmanned Aerial Vehicle” and/or U.S. application Ser. No. 14/844,841filed on 3 Sep. 2015 and titled “System and Methods for Hosting Missionswith Unmanned Aerial Vehicles” which are each incorporated in itsentirety by this reference. For example, the method 100 can beimplemented using one or more UAVs that include system elements such asimage sensors (e.g., cameras) and/or spatial sensors (e.g., GPSreceivers; inertial measurement units (IMUs), such as gyroscopes,accelerometers, and/or magnetometers; altimeters; etc.). However, themethod 100 can additionally or alternatively be implemented using anyother suitable system or system elements.

Block S110 recites: generating a set of mission parameters for a missionof the UAV, the mission of the UAV associated with aerial scanning of aregion of interest. Block S110 is preferably implemented using aplatform interfaced with one or more unmanned aerial vehicles (UAVs) andoperable to generate and execute mission plans, and in embodiments,variations, and examples, is implemented using the platform described inU.S. application Ser. No. 14/717,955 filed on 20 May 2015 and titled“Method for Adaptive Mission Execution on an Unmanned Aerial Vehicle”and/or U.S. application Ser. No. 14/844,841 filed on 3 Sep. 2015 andtitled “System and Methods for Hosting Missions with Unmanned AerialVehicles”.

The region of interest in Block S110 can be a contiguous region ofinterest, or can alternatively be a non-contiguous region of interest(e.g., a region of interest including non-connected bodies). The regionof interest in Block S110 can be user defined (e.g., through anapplication interface that allows a user to select or otherwise defineboundaries of the region of interest). The region of interest in BlockS110 can alternatively be defined in any other suitable manner.

Block S110 preferably includes identifying one or more flight paths forthe UAV that adequately cover the region of interest. The flight pathpreferably includes desired position nodes of the UAV in associationwith one or more of time points and/or UAV states (e.g., velocity,acceleration, heading, system states, etc.). The flight path ispreferably a flight path that is optimized or otherwise designed aroundone or more factors including one or more of: factors associated withthe region of interest (e.g., distance, altitude, terrain features, windconditions, pressure conditions, temperature conditions, precipitationconditions, visibility conditions, etc.), factors associated withoperation of the UAV system(s) (e.g., battery state, control surfacestate, powertrain state, electrical system state, housing state, weight,etc.), factors associated with air traffic/air space, and any othersuitable factor(s). In one variation, the flight path can be theshortest distance flight path covering the region of interest withappropriate heading and/or altitude changes. In a specific example ofthis variation, the flight path comprises a serpentine- orboustrophedonic-shaped flight path including approximately straightsegments (i.e., segments of constant heading/ground track) separated bytransitional segments (e.g., substantially straight transitionalsegments, rounded segments, etc.) coupling a downstream end of a segmentto an upstream end of the following segment (e.g., as shown in FIG. 2);however, the flight path(s) can alternatively be defined in any othersuitable manner (e.g., as a spiraling flight path, etc.).

As such, the flight path is preferably generated in consideration of oneor more of: altitude considerations, heading considerations, anticipatedenvironmental factors (e.g., winds aloft, temperature, pressure,visibility, moisture, bird activity, etc.) at positions of the UAV alongthe flight path, takeoff performance, climb performance, descentperformance, estimated time enroute considerations, power systemrequirements and/or power system state, sensor system/camera systemrequirements (e.g., in terms of environmental conditions) and/or state,control surface state, number of UAVs or other air vehicles associatedwith the flight path, air traffic considerations, air spaceconsiderations, frequency of data collection, frequency of datatransfer, weight and balance of the UAV, payload or other weightconsiderations of the UAV, and any other suitable considerations.

The flight path can be designed or otherwise modified due to changes inconditions, as described in U.S. application Ser. No. 14/717,955 filedon 20 May 2015 and titled “Method for Adaptive Mission Execution on anUnmanned Aerial Vehicle” and/or U.S. application Ser. No. 14/844,841filed on 3 Sep. 2015 and titled “System and Methods for Hosting Missionswith Unmanned Aerial Vehicles”.

Block S120 recites: for each of a set of one or more subregions of theregion of interest, executing a corresponding subportion of the missionwith transmission of the UAV along the flight path (e.g., wherein thesubportion includes an starting phase, an intermediate phase, and aterminating phase). Block S120 functions to execute the mission definedin Block S110, in order to facilitate collection and processing of videodata, telemetry data, and/or any other suitable type of data fromsensors associated with the UAV.

Each subportion of the mission can include a segment of approximatelyconstant or known conditions (e.g., constant heading conditions,constant ground track conditions, constant altitude conditions, constantorientation conditions, etc.) coupled to an upstream segment by a firsttransition segment (e.g., associated with one or more heading changes)and coupled to a downstream segment by a second transition segment(e.g., associated with one or more heading changes). Additionally oralternatively, some or all conditions (e.g., heading, ground track,altitude, orientation, etc.) associated with one or more subportions ofthe mission may be varying and/or unknown. In one embodiment, a missionsubportion includes substantially maintaining a constant altitude,pitch, and roll (e.g., at all times, when not performing maneuvers,etc.), such as substantially zero pitch and roll (e.g., relative agravity vector). In an example of this embodiment, a mission subportion(e.g., the only subportion of the mission) includes following apredefined ground track (e.g., enabling imaging of an entire region ofinterest, preferably while achieving a predetermined front and/or sideoverlap between images), such as a boustrophedonic or spiral groundtrack, while maintaining a substantially constant altitude and, when notturning and/or compensating for changes in external forces, such as windgusts, maintaining a substantially constant pitch and roll. However, themission subportion(s) can additionally or alternatively include anyother suitable segments.

Each subportion of the mission preferably includes collection, storage,and/or transmission of both telemetry data and video data from the UAVfor downstream processing in relation to generation of one or moreaerial maps as facilitated by subsequent blocks of the method 100. Invariations, Block S120 thus comprises real-time and/or near real timetransmission of telemetry data (e.g., GPS data, IMU data, etc.) from theUAV and video data captured using the UAV to associated computingsystems for processing. Each subportion of the mission can additionallyor alternatively include collection, storage, and/or transmission of anyother suitable type of data. Data can be collected and/or transmitted atany suitable rate (e.g., from 5-50,000 Hz). Block S120 preferablyfurther includes registering (e.g., registering in time) the telemetrydataset and the video dataset from the UAV, in order to facilitatedownstream processing steps.

Each subportion of the mission can additionally or alternatively includeone or more of: an acceleration phase, an intermediate phase, and adeceleration phase. The acceleration phase can be associated withacceleration of the UAV from a low-velocity state (e.g. upon exiting aprevious subportion of the mission) to a desired velocity associatedwith the intermediate phase, the intermediate phase can be associatedwith cruise settings of the UAV along a majority of the subportion ofthe mission, and the deceleration phase can be associated withdeceleration of the UAV from the intermediate phase to a low velocitystate associated with transitioning into the next subportion of themission. One or more subportions of the mission can additionally oralternatively include one or more of a climbing phase, a descendingphase, a cruise phase, an upwind phase, a downwind phase, a crosswindphase, and/or any other suitable phase.

In some variations, Block S120 can include a preliminary phase, prior toexecution of the first subportion of the mission, the preliminary phaseincluding transmission of the UAV from a home position to an initiatingposition of the first subportion of the mission defined in Block S110.Block S120 can additionally or alternatively include execution of apre-flight mission according to one or more methods described in U.S.application Ser. No. 14/717,955 filed on 20 May 2015 and titled “Methodfor Adaptive Mission Execution on an Unmanned Aerial Vehicle” and/orU.S. application Ser. No. 14/844,841 filed on 3 Sep. 2015 and titled“System and Methods for Hosting Missions with Unmanned Aerial Vehicles”.

In a specific example, each subportion of the mission covers anapproximately rectangular strip of the region of interest, such that,collectively, the subportions cover approximately parallel subportionsof the region of interest. In the example, adjacent strips associatedwith each subportion of the mission are preferably overlapping (inrelation to downstream Blocks of the method 100 associated with assemblyof the strips into a representation of the region of interest).Alternatively, adjacent strips associated with each subportion of themission can be non-overlapping (e.g., directly abutting, includingcoverage gaps between the strips, etc.). As such, in a first approach,each frame is added to a strip, and then strips are assembled to form amosaic (e.g., wherein the first strip formed becomes the base of themosaic, and subsequent strips are added to the mosaic). In analternative second approach (e.g., in which the mission includes only asingle subportion, in which the mission subportions are associated withnon-contiguous or non-overlapping regions, etc.), each incoming framecan be directly added to a base mosaic (without requiring assembly ofstrips assembled from frames of data). As such, some variations of themethod 100 can omit portions of Block S130 and/or S140, in relation togenerating subassemblies from frames of incoming data. In an alternativethird approach, mosaic assembly can be performed by assembling a mix ofincoming frames and strips of assembled frames. As such, assembly of afinal mosaic can be performed using a combination of techniques and/orin any other suitable manner.

Block S130 recites: during each subportion of the mission, generating animage subassembly corresponding to the subportion of the mission. BlockS130 functions to generate (e.g., iteratively generate) subportions of acomplete representation (e.g., orthomosaic map), preferably includingmonitoring of UAV state/data quality and/or pre-processing steps thatcan facilitate rapid assembly of data from the UAV into a completerepresentation of the region of interest. The image subassemblies can beapproximately rectangular (e.g., as in rectangular strips), or can haveany other suitable morphology, depending upon the flight path of theUAV, camera parameters, and/or any other suitable parameters.

Block S130 preferably implements a subassembly generation algorithm thatprocesses incoming frames of data (e.g., image data, telemetry data,etc.) from the UAV with one or more filtering algorithms associated withUAV state and data quality state.

In one variation, as shown in FIG. 3, Block S130 can include one or moreof: determining if the frame is associated with the starting phase, theintermediate phase, or a terminating phase of a subportion of themission based upon a state rejection filter S131, and/or implementing aframe rejection filter S132 (e.g., before, after, and/or concurrent withS131). Outputs of processing according to S131 and/or S132 can producean image subassembly associated with the corresponding subportion of themission of the UAV. Outputs of Block S132 can then be passed through asubassembly validation step S133 configured for performing a final checkof an assembled subassembly, outputs of which are passed to Block S140for combining a validated subassembly with a running aggregation ofimage sub-assemblies.

In an example, Block S131 includes passing an incoming frame through amodule that determines if a subassembly is currently being recorded.Upon determination that a subassembly is not being recorded, the modulethen initiates recording of a new subassembly. However, upondetermination that a subassembly is being recorded, the module continuesrecording according to the state rejection filter of Block S131, untilrecording of the subassembly is complete. In one variation, Block S131can include determining if pixels of an incoming frame correspond withthe starting phase, the intermediate phase, or the terminating phase ofa mission subportion, based upon features extracted from the telemetrydataset (e.g., based upon UAV heading state, based upon UAV positionstate, based upon other UAV state, etc.). For instance, upon receivingan incoming frame from the UAV, Block S131 can comprise extracting aheading of the UAV from the telemetry dataset, and if the heading isassociated with a intermediate phase of a subportion of the mission,initiation of recording of a new subassembly will not occur. Similarly,upon receiving an incoming frame from the UAV, Block S131 can compriseextracting a heading of the UAV from the telemetry dataset, and if theheading is associated with a transitional phase of a subportion of themission, initiation of recording of a new subassembly will occur.However, Block S131 can additionally or alternatively be implemented inany other suitable manner (or Block S131 can alternatively be omitted).

In addition to heading of the UAV, the state rejection filter of BlockS131 can additionally or alternatively process position in relation toterrain, as shown in FIG. 4, in order to facilitate related frameprocessing steps (e.g., pixel selection, frame alignment, frame skewing,etc.). In more detail, absolute altitude above specific terrain features(e.g., falling terrain, rising terrain) can produce image frames thatrequire additional frame processing steps, so that the frame can beblended appropriately with adjacent frames. Thus, implementation of thestate rejection filter of Block S131 can include identifying position ofthe UAV relative to specific terrain features, in order to guidesubsequent frame processing steps associated with pixel selection, framealignment, frame blending, and/or any other suitable aspect ofgenerating the subassembly from frames.

Additionally or alternatively, the state rejection filter of Block S131can perform an assessment of one or more orientations of the UAV, inrelation to one or more of pitch, roll, and yaw, in order to guidesubsequent frame processing steps associated with pixel selection, framealignment, frame blending, and/or any other suitable aspect ofgenerating the subassembly from frames. Additionally or alternatively,the state rejection filter of Block S131 can perform an assessment ofone or more of: UAV video and/or image feed states (e.g., in relation toimage artifacts, image discoloration, image blur, compression artifacts,etc.), UAV energy management system states, UAV powertrain systemstates, states of sensors of the UAV, states of instruments of the UAV,states of control surfaces of the UAV, states of the environment and/orclimate (e.g., in terms of visibility, moisture, winds, temperature,pressure, environmental lighting, etc.), and any other suitable states.

The frame rejection filter of Block S132 determines if a frame should beincorporated into the subassembly, or if further processing of the frameinto the subassembly should not occur, and instead, a new incoming frameshould be processed. In an example, as shown in FIG. 3 implementing theframe rejection filter of Block S132 can include a pixel selectionoperation S1321, a frame alignment operation S1322, and a frame blendingoperation S1323. Block S132 can additionally or alternatively include aframe corruption check operation and/or a frame preprocessing operation.Block S132 can, however, additionally or alternatively include any othersuitable operations to determine if a frame has suitable quality forblending into a current subassembly.

The frame corruption check operation functions to determine whether dataassociated with the frame is corrupted (wherein corrupted frames arepreferably ignored and/or discarded). The corruption check is preferablyperformed before any other frame processing and/or analysis, but canadditionally or alternatively be performed at any other suitable time.The corruption check can include, for example, examining data headers(e.g., to determine conformance to header standards, wherein lack ofconformance can indicate corruption), calculating hashes, and/orexamining checksums (e.g., wherein a calculated hash value or checksumvalue not equal to an expected value can indicate corruption). Forexample, the image data can include an H264 video stream, and the framecorruption check operation can include examining the checksum of a rawH264 frame of the stream. However, the frame corruption check operationcan additionally or alternatively be performed in any other suitablemanner.

The frame preprocessing operation preferably functions to prepare aframe for analysis, modification, and/or storage (e.g., for subsequentoperations of Block S132). The frame preprocessing operation can includeone or more of: decoding and/or decompressing the frame, converting theframe to greyscale, downsampling the frame, and blurring the frame. Theframe is preferably stored in (and/or converted to) a luma-based (orluminance-based) color space (e.g., YUV color space), such that the lumachannel can be used directly (e.g., obviating the need for conversion togreyscale). In one example, downsampling and/or blurring (e.g.,performing a Gaussian blur operation) can be performed using optimizedhardware vector operations, such as an ARM NEON routine. However, theframe preprocessing operation can additionally or alternatively includeany other operations performed in any suitable manner.

The pixel selection operation of Block S1321 functions to select areduced or minimized number of pixels from the incoming frame, in orderto reduce processing load for downstream blocks of the method 100. Thepixel selection operation can implement an algorithm in computer codeoperable to identify regions of pixels associated with inappropriateposition and/or velocity parameters of the UAV (e.g., pixels at the edgeof a frame) that do not need to be processed in downstream steps (e.g.,in a pixel filtering operation), and passes references to selectedpixels (e.g., indices of selected pixels, regions of selected pixels,etc.) to subsequent blocks of the method 100. Selected pixels caninclude pixels to be analyzed (e.g., for feature tracking), pixels to beblended into the subassembly, and/or any other suitable pixels. In oneembodiment, Block S1321 includes selecting and/or assessing bands ofpixels (e.g., horizontal bands, such as bands substantially normal anaircraft velocity vector at the time of image capture). For example,Block S1321 can include (e.g., after the feature extraction operation ofBlock S321) determining a quality metric for each band of a frame, suchas based on statistical metrics associated with the extracted features(and/or scores associated with the features, such as corner scores)and/or the pixel values, wherein only bands above a threshold qualitymetric are selected. Block S1321 can thus omit unnecessary pixels fromfurther processing. However, Block S1321 can additionally oralternatively be performed in any other suitable manner, or canalternatively be omitted.

The frame alignment operation of Block S1322 functions to perform acombination operation with pixels from multiple incoming frames, whereinthe combination operation includes one or more of stacking and blendingof pixels from multiple frames. The frame alignment operation of BlockS1322 can additionally or alternatively include operations associatedwith skewing (e.g., transforming a rectangular array of pixels into atrapezoidal array, transforming an array of pixels into an array ofanother shape, etc.) of frames (and/or subsets thereof) in relation tothe state rejection filter of Block S131. For instance, in relation toprocessing a frame associated with rising terrain, the frame alignmentoperation of Block S1322 can transform the rectangular array of pixelsinto a skewed array of pixels in order to facilitate blending of theframe into the image subassembly.

Block S1321 and S1322 can be implemented in any suitable order. Forinstance, pixel selection can be performed prior to alignment, oralignment can be performed prior to pixel selection. Furthermore, BlocksS1321 and S1322 can be implemented in coordination with one or morecomputer vision techniques including feature tracking techniques S321and/or optimal transform estimate techniques S322. As shown in FIG. 3,outputs of the pixel selection operation can be provided to a computervision module operable to perform feature tracking operations on pixels,in order to track features of interest from each incoming frame tofacilitate assembly of the incoming frames. The feature trackingoperation of Block S321 can include one or more of edge detection,corner detection, blob detection, ridge detection, and any other featuredetection process, which in specific examples can include or be derivedfrom one or more of: a Canny approach, a Sobel approach, a Kayyaliapproach, a Harris & Stephens/Plessey/Shi-Tomasi approach, a SUSANapproach, a level curve curvature approach, a FAST approach, a Laplacianof Gaussian approach, an MSER approach, a PCBR approach, and any othersuitable approach.

Block S321 can include a feature extraction operation and/or a featurematching operation. The feature extraction operation preferablyfunctions to determine (e.g., using techniques such as those describedabove) a set of features (e.g., corners, edges, blobs, ridges, featurescorresponding to physical features of the imaged region, etc.), eachfeature associated with a different subset of the frame (“photosubset”). The photo subset can be a set of pixels, an image region, acontiguous photo subset, and/or any other suitable subset of the frame.Each feature extracted is preferably of the same feature type (e.g., allcorners, all blobs, etc.), but the set of features can alternativelyinclude features of multiple types. However, the feature extractionoperation can additionally or alternatively be performed in any othersuitable manner to determine any suitable number and/or type offeatures.

In one embodiment, the feature extraction operation includes subdividingthe frame into multiple segments and extracting features from eachsegment. Features can then be extracted from each segment of the frame(e.g., extracting an equal or similar number of features from eachsegment, extracting features in numbers substantially proportional tosegment area, etc.). Such subdivision can function to achieve a moreeven distribution of features across the frame. Preferably, the segmentsare non-overlapping and cooperatively span the frame, such as defining agrid (e.g., rectangular grid of equivalently-shaped segments) coveringthe frame, but can alternatively exclude any suitable frame regionsand/or overlap in any suitable manner. For example, the featureextraction operation can include: segmenting the frame into a set ofphoto segments cooperatively defining a grid; determining a targetfeature number; and for each photo segment of the set, determining arespective subset of features consisting essentially of a number offeatures equal to the target feature number, wherein each feature of thesubset is depicted by the photo segment (e.g., wherein the set offeatures consists essentially of the respective subset of features ofeach photo segment).

The feature matching operation preferably functions to determine matchesbetween features extracted from different frames (e.g., using thetechniques described above and/or any other suitable techniques). Thefeature matching operation can include determining (e.g., for some orall of the features of the set extracted from a particular frame)matches with features extracted from other frames (e.g., previouslycaptured and/or processed frames). Determining such matches preferablyincludes determining the associated photo subset of each frame thatdepicts the feature, wherein a match indicates that the photo subsetsare believed to depict the same physical feature in the imaged region.The feature matching operation can optionally include eliminatingoutlier matches, such as by a random sample consensus (RANSAC) process.

The extracted and/or matched feature information determined in BlockS321 can be used in Block S1321, in Block S322, in Block S1322, and/orin any other suitable operations of the method. For example, the matchedsets of features can be output to the optimal transform estimateoperation for use as described below.

Also shown in FIG. 3, an optimal transform estimate operation S322 canimplement computer vision techniques (e.g., for determining featurelocations within the region, for transforming an incoming frame into askewed and/or otherwise transformed frame, etc.). Block S322 caninclude, for example, one or more of: a triangulation operation, abundle adjustment operation, and a transform optimization operation.

The triangulation operation preferably functions to estimate thepositions of the physical features (e.g., extracted and matchedfeatures) within the region, such as by back-projecting matched featuresinto 3D space to determine an associated physical feature position in 3Dspace (e.g., as shown in FIG. 6). For example, the triangulationoperation can include, for each matched feature (e.g., from Block S321),determining a position including a three-dimensional coordinate setrepresentative of a location within the imaged region (e.g., a latitude,longitude, and altitude; a set of displacement coordinates relative to areference location, such as a location within the region; etc.). Thetriangulation operation preferably uses data indicative of cameraposition and/or orientation (e.g., telemetry data, such as UAV GPSand/or IMU data) to determine (e.g., estimate) the camera pose (positionand orientation) for triangulation, but can additionally oralternatively determine the camera pose in any other suitable manner.

The bundle adjustment operation preferably functions to refine theestimates of the triangulation operation, such as by minimizing thereprojection error associated with the determined physical featurepositions. The bundle adjustment operation can include optimizing (e.g.,to minimize the reprojection error) over one or more of: cameraintrinsics parameters (e.g., focal length, lens distortion coefficients,etc.), camera extrinsics parameters (e.g., position, orientation), andphysical feature positions. The bundle adjustment is preferablyperformed for each new frame, but can alternatively be performed foronly a subset thereof, or be performed any other suitable number oftimes. However, the method can alternatively omit the bundle adjustmentoperation (e.g., wherein subsequent operations, such as the transformoptimization operation, rely instead on the estimated spatial featurepositions determined by the triangulation operation and/or the cameraposes determined based on telemetry data).

Optimization over some or all parameters during bundle adjustment canoptionally be restricted (e.g., to reduce processing requirements).Optimization restrictions can include fixing parameters at specificvalues and/or constraining parameters to remain within a specific rangeof values. Such restrictions are preferably imposed on camera poseparameters, more preferably altitude, pitch, and/or roll. However,restrictions can additionally or alternatively be imposed upon otherpose parameters, such as forward motion (e.g., motion along a plannedheading or ground track), lateral motion (e.g., motion normal theplanned planned heading or ground track), and/or yaw, and/or can beimposed on any other suitable parameters. In a first variation, thealtitude, pitch, and roll of each camera pose are fixed at specificvalues during bundle adjustment. In a first example of this variation,the values are equal for each frame (constant altitude, pitch, androll), corresponding to a planar constraint. In a second example, thealtitude, pitch, and roll of each camera pose are fixed at valuescorresponding to the telemetry data collected at (or near) that pose. Ina second variation, the altitude, pitch, and roll of each camera poseare constrained within a range around the initial estimated values(e.g., values used in the triangulation process). For example, eachaltitude value can be constrained within a threshold distance (e.g., 0.1m, 0.25 m, 0.5 m, 1 m, 2 m, 3 m, 5 m, 10 m, etc.) of the correspondinginitial estimate, and each pitch and roll value can be constrainedwithin a threshold angular distance (e.g., 1°, 3°, 5°, 10°, 15°, 20°,30°, etc.) of the corresponding initial estimate. In a third variation,a subset of bundle adjustments (e.g., one out of each set of consecutiveframes and/or bundle adjustments, such as one out of every 3, 5, 7, 10,25, 100, 2-10, 10-30, or 30-100 frames) are performed withoutoptimization restrictions (or with fewer restrictions, such as usingconstrained ranges rather than fixed values), and the values determinedin the unoptimized bundle adjustments are used for the restricted bundleadjustments of nearby frames. In one example, an unrestricted bundleadjustment is performed once every 7 frames, and the following bundleadjustments (corresponding to the following 6 frames) are performedfixing the altitude, pitch, and roll based on the values determined inthe unrestricted bundle adjustment (e.g., set equal to the determinedvalues, extrapolated based on changes in the determined values, etc.).However, the optimizations can additionally or alternatively beconstrained in any other suitable manner, or can alternatively beunconstrained.

The data used for each iteration of bundle adjustment can optionally berestricted. In some embodiments, each bundle adjustment process (e.g.,corresponding to each processed frame) can be performed based on alimited number of frames (e.g., limited to 3, 5, 7, 10, 15, 25, 2-20, or20-200 frames, in addition to or including the current frame), which canfunction to reduce processing requirements and maintain substantiallyconstant processing requirements throughout performance of the method(e.g., despite increasing numbers of collected and/or processed frames).In a first example, the frames used for the bundle adjustment arerestricted based on a temporal window (e.g., selecting the most recentlycaptured and/or processed frames). In a second example, frames arerestricted based on a spatial window (e.g., frames with the closestcamera positions, the most overlapped view regions, and/or the greatestnumber of matched features). In a third example, a random subset offrames (e.g., of all captured and/or processed frames, of frames withoverlapping view regions and/or matched features, etc.) is used.However, frame usage can additionally or alternatively be limited in anyother suitable manner, or can alternatively be unlimited.

The transform optimization operation preferably functions to determineoptimized transforms to apply to the frames (or portions thereof), suchas by rectifying (e.g., orthorectifying) the frames, thereby enablingthe frames to be blended (e.g., to generate an orthomosaic map). Thetransform optimization operation is preferably performed based on thespatial feature positions and/or camera poses determined previously(e.g., determined by the bundle adjustment operations and/or thetriangulation operations), but can additionally or alternatively beperformed based on any other suitable information.

In one specific example, Block S322 can estimate optimal frame alignmentand/or transformation (e.g., affine transformation) as a correction to atelemetry-based approach for frame alignment, but if the computer visionapproach(es) fail(s), the telemetry-based approach would be implementedas an un-optimized default (e.g., in relation to processing dataassociated with homogenous terrain or constantly changing terrain, suchas a body of water). However, computer vision techniques canadditionally or alternatively be implemented in coordination withtelemetry-based approaches for omitting pixels from frames and/oraligning frames.

The frame blending operation of Block S1323 functions to blend aprocessed incoming frame into a running aggregation of assembled frames(e.g., in order to create the image subassembly associated with asubportion of a mission, in order to create a running aggregation of allprocessed images of the mission, etc.). The frame blending operation canimplement any one or more of: seam reduction algorithms (e.g. using gainadjustment approaches proximal to the seam), algorithms to mitigateparallax, algorithms to mitigate distortion effects (e.g., due to thecamera lens, due to rolling shutter effects, etc.), algorithms tomitigate exposure/lighting effects (e.g., in relation to environmentallighting changes), and any other suitable algorithms.

The frame blending operation can include generating a blended imageand/or image pyramid (e.g., Laplacian pyramid, Gaussian pyramid,steerable pyramid, etc.). The blending weights can optionally berepresented as (e.g., stored as, computed as, etc.) integer values(e.g., 8 bit integers, 16 bit integers, etc.), which can reduce storage,memory, and/or processing requirements as compared with using floatingpoint representations (e.g., 32 bit floats, 64 bit doubles, etc.), butcan additionally or alternatively be represented using floating pointrepresentations and/or any other suitable representations. The pyramidlevels (e.g., of a Laplacian image pyramid) can be represented ascompressed low-precision integers (e.g., signed 8 bit integers), whichcan reduce requirements as compared with using higher-precisionrepresentations (e.g., signed i6 bit integers), but can additionally oralternatively be represented using higher-precision representationsand/or any other suitable representations.

Further optimizations (e.g., reducing processing and/or storage usage)can optionally be employed regarding storage and/or output of thegenerated images and/or image pyramid. For example, an image (e.g.,pyramid tile, blended image subset, etc.) can optionally be stored(e.g., uncompressed, lightly compressed, etc.) in memory until after athreshold time has elapsed without the image being modified (e.g., bysubsequent bundle adjustment, transform, and/or blending operations),after which the image can be compressed (e.g., converted to a PNGrepresentation). Additionally or alternatively, the image (e.g.,uncompressed, lightly compressed, finally compressed such as PNG, etc.)can be retained in memory (e.g., and not written to long-term storage)until after a second threshold time (e.g., longer than the firstthreshold time for image compression) has elapsed without the imagebeing modified, after which the image can be written to long-termstorage. However, the frame blending operation can additionally oralternatively employ any other suitable optimizations, and/or can beperformed in any other suitable manner.

Block S140 recites: during each subportion of the mission, combining theimage subassembly with a running aggregation of image subassembliesassociated with other subportions of the mission. Block S140 functionsto process incoming subassemblies from Block S130 and to blend them witha running aggregation of image assemblies, but, as indicated above, canalternatively be substituted with blocks of steps associated withaddition of incoming frames to a base mosaic (without aggregation ofstrips or other image assemblies). As such, a subassembly can includeany suitable number of frames. Block S140 preferably includes blocksassociated with subassembly re-orientation, subassembly transformation,subassembly blending with the running aggregation (i.e., an orthomosaicor an orthomosaic-in-progress), canvas adjustment, and any othersuitable step. The running aggregation is preferably represented as oneor more blended images and/or image pyramids, such as an image pyramidgenerated from the blended images and/or image pyramids generated inBlock S1323, but can additionally or alternatively be represented in anyother suitable manner.

In one variation, as shown in FIG. 3, Block S140 can include Block S141,which recites: re-orienting an incoming subassembly, if needed, to matchthe orientation of the running aggregation of subassemblies. In theexample involving a serpentine or boustrophedonic flight path, BlockS141 can include rotating, by heading, alternating incomingsubassemblies (e.g., rotating by approximately 180°). However, dependingupon the flight path, subassemblies can alternatively be rotated by anysuitable number of degrees to align the orientations of all of theincoming subassemblies.

As shown in FIG. 3, Block S140 can additionally or alternatively includeBlock S142, which recites: performing a transformation operation on theincoming subassembly. Similar to the pixel selection and frame alignmentoperation described in relation to Blocks S131-S132, the transformationoperation 142 can be implemented without computer vision techniques, orcan alternatively be implemented using one or more computer visiontechniques. In one variation, Block S142 can include an optimaltransform estimate operation S1421, which processes feature detectioninputs and performs feature mapping of features across differentsubassemblies, in order to perform the optimal transform estimateoperation. In a specific example, Block S1421 can compute an optimalaffine transformation between multiple point sets (e.g., two 2D pointsets, any suitable number of point sets in any number of dimensions).Similar to Blocks S131-S132, if the computer vision enhanced operationsfail, Block S142 can include defaulting to translating pixels of anincoming subassembly based on known position differences betweenadjacent subassemblies (e.g., differences in latitude, difference inlongitude).

Finally, this variation of Block S140 can include adjusting an imagecanvas associated with the running aggregation of image subassemblies,with the transformed version of the incoming subassembly, and thenperforming a transformation of the running aggregation, if needed.Alternatively (e.g., in embodiments in which the mission includes only asingle subportion), any or all elements of Block S140 can optionally beomitted.

Block S150 recites: during each subportion of the mission, rendering arepresentation of a running aggregation of data associated with themission. The data representation can include: the image subassemblyand/or the running aggregation of image subassemblies; a point cloud(e.g., 2D point cloud, 3D point cloud, etc.), such as a point cloudassociated with the spatial feature positions; a projected map (e.g.,such as projected maps described below regarding Block S170); and/or anyother suitable representation.

The representation is preferably rendered at a display of an electronicdevice of a user, but can additionally or alternatively be rendered atany other suitable display element. Block S150 functions to update arendering of the region of interest, in progress, to a user (e.g., innear-real time; within a threshold time after image capture, such aswithin 0.1 s, 0.3 s, 1 s, 2 s, 3 s, 5 s, 10 s, 15 s, 20 s, 30 s, 60 s,0.1-1 s, 1-10 s, 10-100 s, etc.), such that the user can monitorprogress of scanning of the region of interest. As such, in relation toU.S. application Ser. No. 14/717,955 titled “Method for Adaptive MissionExecution on an Unmanned Aerial Vehicle”, which is incorporated by thisreference, the method 100 can be used to adjust mission parameters or apath of the UAV(s) involved (e.g., after a high-level or low-levelquality scan of the region of interest has been conducted). Furthermore,such information can be used to drive movement and capturing oflower-altitude and/or higher quality image representations of certainareas of interest associated with the region of interest (e.g., based ondetected features of the region of interest, user inputs, etc.).

For example, in response to receiving a refinement request (e.g.,associated with a subset of the region of interest) from the user, themethod can include: modifying the mission parameters based on therequest; capturing additional image frames (e.g., associated with thesubregion); processing the image frames (e.g., as described above, suchas regarding Blocks S130 and/or S140); and displaying the updatedrendering at the electronic device. The refinement request can include auser selection of a subset of the region of interest, a user indicationof a poorly- and/or inadequately-imaged portion of the region ofinterest, a user selection of a “refine region” button, and/or any othersuitable user input or other request. Modifying the mission parameterspreferably functions to control the UAV (and/or a different UAV) to flyover the subregion to be refined (e.g., in a boustrophedonic or spiralpattern), such as at a lower altitude, slower velocity, and/or differentheading (e.g., substantially perpendicular the original heading) ascompared with an already-performed portion of the mission (e.g., earlierUAV flyover of the subregion). This can enable re-imaging of thesubregion, preferably with higher quality than the initial imaging.However, user inputs can additionally or alternatively be treated in anyother suitable manner.

In variations, Block S150 can include rendering the running aggregationat one or more of: a mobile computing device, a personal computingdevice, and/or a wearable computing device (e.g., head-mounted wearablecomputing device, forearm-borne mobile computing device, etc.) of theuser. In variations, Block S150 can additionally or alternativelyinclude rendering the running aggregation at multiple devices of thesame user or different users. In a specific example, shown in FIG. 5,Block S150 includes rendering the running aggregation within anapplication executing at a mobile computing device (e.g., tablet, mobilephone, etc.; preferably the device performing Blocks S130 and/or S140but additionally or alternatively any other suitable devices) of theuser, wherein the mobile computing device is coupled (e.g., by WiFi, byanother wireless communication protocol) to the UAV performing thescanning. Block S150 is preferably performed using the blended image(s)and/or image pyramid(s) generated as described above (e.g., regardingBlock S130 and/or S140), such as by displaying image tiles of the imagepyramid. However, Block S150 can additionally or alternatively beimplemented in any other suitable manner.

As noted above, the method 100 can additionally or alternatively includegenerating an orthorectified representation of the region of interestS160 and/or generating a projected map of the region of interest S170.The orthorectified representation can be a processed version of thefinal iteration of subassembly generation from Blocks S130 and/or S140,or can alternatively be generated in any other suitable manner.Furthermore, the projected map of the region of interest can include oneor more of: an elevation map (e.g., 2.5D representation, topographicmap, etc.), a terrain model, a 3D model, a point cloud (e.g., 3D pointcloud, 2D point cloud, etc.), and any other suitable multidimensionalrepresentation.

The orthorectified representation and/or projected map are preferablygenerated based on the information sampled and/or determined in themethod (e.g., during any or all of Blocks S120-S150), such as based onthe spatial feature positions, camera poses, telemetry data, and/orimages (e.g., captured image frames, orthorectified images, etc.). Theorthorectified representation and/or projected map can be generated bythe same device(s) that perform any or all of Blocks S120-S150,generated by different devices (e.g., by a remote computing system, suchas an internet-connected server), and/or generated in any other suitablemanner. In one example, information determined in Blocks S120-S150, suchas the spatial feature positions, camera poses, telemetry data, and/orimages, is transmitted to a remote computing system, which generates anorthorectified representation (e.g., of greater accuracy than theorthorectified representation generated in Blocks S120-S150, such asgenerated by a user device) and/or projected map by performing one ormore of the following operations: densifying the feature set byextracting and matching additional features (e.g., by tracing alongepipolar lines to find additional pixel matches between differentframes), performing additional bundle adjustments (e.g., unrestrictedbundle adjustments, bundle adjustments using the densified feature set,etc.), determining meshes, determining mesh textures, generatingorthorectified representations and/or projected maps based on thedetermined information, and/or any other suitable processing operations.In some embodiments, the orthorectified representation and/or projectedmap is displayed to the user (e.g., analogously to the renderingdescribed above regarding Block S150, such as shown in FIG. 5; at adifferent device and/or a later time; etc.). However, the orthorectifiedrepresentation and/or projected map can additionally or alternatively beused in any other suitable manner.

The method 100 can additionally or alternatively include Block S180,which recites: generating an analysis from at least one of theorthorectified representation and the projected map of the region ofinterest. Block S180 can include performing photogrammetry-relevantmeasurements and/or providing outputs in terms of distance measurements,area measurements, volume measurements, and/or any other suitablemeasurements from the region of interest. Block S180 can additionally oralternatively include performing analyses of homogenous terrain, and/oridentification of anomalies within mapped homogenous terrain (e.g., inthe context of rescue missions, etc.). Block S180 can additionally oralternatively include performing graphical annotations of the region ofinterest according to known or custom indices (e.g., in relation toplant health analysis (e.g., NDVI Analysis, VARI analysis, OSAVIanalysis, etc.), in relation to other normalized indices, in relation toother enhanced indices, etc.). Block S180 can additionally oralternatively include performing automated map cropping and/or otherimage analysis techniques to facilitate observation of the region ofinterest. Block S180 can additionally or alternatively includeimplementing thermal analyses of the region(s) of interest, in relationto identifying areas associated with differences in temperature fromexpected temperature profiles. Such applications of thermal analyses canbe used for search and rescue missions, livestock counting/livestockpopulation analyses, population analysis associated with otherorganisms, and/or any other suitable type of analysis. Block S180 can beperformed by the same device(s) that perform any or all of BlocksS120-S170, generated by different devices (e.g., by a remote computingsystem, such as an internet-connected server), and/or generated in anyother suitable manner. However, Block S180 can additionally oralternatively include generation of any other suitable analysis.

In some variations (e.g., as shown in FIG. 7), the method includes(e.g., in Block S130), for each image frame captured during missionperformance (or a subset thereof), one or more of: a state rejectionoperation; a frame rejection operation; a pixel selection operation; afeature extraction operation; a feature matching operation; atriangulation operation; a bundle adjustment operation; a transformoptimization operation; and a frame blending operation. However, themethod 100 can additionally or alternatively include any other suitableelements performed in any suitable manner.

While the method 100 above is described in the context of one or alimited number of UAVs, the method 100 can additionally or alternativelybe implemented using multiple UAVs. For instance, the method 100 canimplement a UAV fleet configured to sweep the region of interest,wherein each UAV has jurisdiction over at least one subportion of theregion of interest, follows a flight path over its jurisdiction(s), andthen modified frame processing and/or images subassembly processingoccurs according to variants of Blocks S130-S140 above. Furthermore, ifone UAV in the fleet is incapable of and/or delayed in executing and/orcompleting its mission (e.g., the UAV is disabled, the UAV performsportions of the mission more slowly than expected, etc.), one or moreother UAVs in the fleet can perform the mission of the incapable UAV(e.g., upon completion of its own mission). However, multiple UAVs canbe configured in any other suitable manner to perform variations of themethod 100.

The methods and/or systems of the invention can be embodied and/orimplemented at least in part in the cloud and/or as a machine configuredto receive a computer-readable medium storing computer-readableinstructions. The instructions can be executed by computer-executablecomponents integrated with the application, applet, host, server,network, website, communication service, communication interface,hardware/firmware/software elements of a user computer or mobile device,or any suitable combination thereof. Other systems and methods of theembodiments can be embodied and/or implemented at least in part as amachine configured to receive a computer-readable medium storingcomputer-readable instructions. The instructions can be executed bycomputer-executable components integrated by computer-executablecomponents integrated with apparatuses and networks of the typedescribed above. The computer-readable medium can be stored on anysuitable computer readable media such as RAMs, ROMs, flash memory,EEPROMs, optical devices (CD or DVD), hard drives, floppy drives, or anysuitable device. The computer-executable component can be a processor,though any suitable dedicated hardware device can (alternatively oradditionally) execute the instructions.

The FIGURES illustrate the architecture, functionality and operation ofpossible implementations of systems, methods and computer programproducts according to preferred embodiments, example configurations, andvariations thereof. In this regard, each block in the flowchart or blockdiagrams may represent a module, segment, step, or portion of code,which comprises one or more executable instructions for implementing thespecified logical function(s). It should also be noted that, in somealternative implementations, the functions noted in the block can occurout of the order noted in the FIGURES. For example, two blocks shown insuccession may, in fact, be executed substantially concurrently, or theblocks may sometimes be executed in the reverse order, depending uponthe functionality involved. It will also be noted that each block of theblock diagrams and/or flowchart illustration, and combinations of blocksin the block diagrams and/or flowchart illustration, can be implementedby special purpose hardware-based systems that perform the specifiedfunctions or acts, or combinations of special purpose hardware andcomputer instructions.

As a person skilled in the art will recognize from the previous detaileddescription and from the figures and claims, modifications and changescan be made to the embodiments of the invention without departing fromthe scope of this invention as defined in the following claims.

We claim:
 1. A system comprising: a mobile user device communicativelyconnected to a camera and configured to receive images of a geographicregion from the camera, the mobile device comprising a memory and aprocessor defining a processing capacity, wherein the processor isconfigured to, for each image: determine a set of features for theimage; determine a window of images, within a set of previous images ofthe geographic region, based on the processing capacity, wherein the setof previous images are captured by the camera and stored at the memoryof the mobile user device; and based on the window, determine a refinedcamera pose associated with the image and a set of feature positions,comprising: for each feature of the set of features, determining a setof matches within the window of previous images; based on the set ofmatches, determining a set of estimated positions; and performing abundle adjustment to refine the set of estimated positions.
 2. Thesystem of claim 1, wherein the processor is further configured todetermine a terrain homogeneity score based on a size of the set offeatures, wherein the window of images is determined based on both theprocessing capacity and the terrain homogeneity score.
 3. The system ofclaim 1, wherein the processor is further configured to determine acamera pose estimate based on telemetry data associated with the image,wherein the window comprises a spatial window, wherein the spatialwindow is determined based on the camera pose estimate.
 4. The system ofclaim 1, wherein the processor is further configured to: generate anorthomosaic map based on the set of feature positions; and provide theorthomosaic map at a display of the mobile user device.
 5. The system ofclaim 4, further comprising an aircraft, the camera located onboard theaircraft; wherein the orthomosaic map is provided at the display duringaircraft flight, wherein the mobile user device is configured to:receive user control inputs based on the orthomosaic map and wirelesslycontrol the aircraft based on the user control inputs.
 6. The system ofclaim 5, wherein the wireless control of the aircraft comprisesmodifying parameters of an aircraft mission during execution of themission.
 7. The system of claim 4, wherein the processor is furtherconfigured to generate a secondary inspection plan for the geographicregion based on the orthomosaic map.
 8. The system of claim 7, whereinthe inspection plan comprises a mission plan for a secondary flight ofan aircraft.
 9. The system of claim 4, wherein the processor is furtherconfigured to: generate an analysis based on the orthomosaic map; and atthe display, provide a graphical annotation of the orthomosaic map basedon the analysis.
 10. The system of claim 9, wherein the analysiscomprises a population analysis for a group of organisms within thegeographic region.
 11. The system of claim 1, wherein performing thebundle adjustment to refine the set of estimated positions comprisesminimizing a reprojection error of the set of estimated positions,wherein the reprojection error is not minimized with respect to a subsetof camera position parameters.
 12. A method comprising: at a mobile userdevice, receiving an image from a camera at a receipt time; determininga set of features based on the image; determining a sliding window ofaerial images within a set of historical images from the camera storedat the mobile user device, wherein a size of the sliding window isdetermined based on a scene homogeneity for the image and a processingcapacity of the mobile user device; for each feature of the set offeatures, determining a set of matches within the sliding window ofaerial images, determining a set of feature positions for each featureof the set of features; at the mobile user device, generating anorthomosaic image based on the set of feature positions; and at themobile user device, displaying the orthomosaic image in real timerelative to the receipt time.
 13. The method of claim 12, wherein thecamera is wirelessly connected to the mobile user device.
 14. The methodof claim 13, wherein the camera is arranged onboard an aircraft which iscontrolled by the mobile user device based on the orthomosaic image. 15.The method of claim 14, further comprising, at the mobile user device,modifying mission parameters of a mission for the aircraft duringexecution of the mission.
 16. The method of claim 12, furthercomprising: at the mobile user device, generating a mission plan for animaging aircraft based on the orthomosaic image.
 17. The method of claim12, further comprising: generating an analysis based on the orthomosaicimage; and graphically annotating the orthomosaic image based on theanalysis.
 18. The method of claim 17, wherein the analysis comprises aplant health analysis.
 19. The method of claim 12, wherein theorthomosaic image comprises a 2D representation of a scene captured bythe set of historical aerial images.
 20. The method of claim 12, whereinthe sliding window is determined based on at least one of: a spatialwindow associated with a camera position for the image or a temporalwindow associated with an image timestamp.